Isolated Unitary Representations

In this paper we collect some facts about the topology on the space of irreducible unitary representations of a real reductive group. The main goal is Theorem 10, which asserts that most of the “cohomological” unitary representations for real reductive groups (see [VZ]) are isolated. Many of the intermediate results can be extended to groups over any local field, but we will discuss these generalizations only in remarks. The foundations of the topological theory of the unitary dual are actually more easily available in the nonarchimedean case, particularly in the work of Tadić [Tad1] and [Tad3]. In the archimedean case the best results are due to Miličić, but unfortunately only part of this has been published in [Mil] and [Mil3]. Suppose then that G is a real reductive Lie group. Write Ĝ for the unitary dual of G. The Fell topology on Ĝ is defined as follows. Suppose S ⊂ Ĝ. An irreducible unitary representation π belongs to the closure of S if and only if every matrix coefficient (equivalently, a single non-zero matrix coefficient) of π is the uniform limit on compact sets of matrix coefficients of elements of S. A convenient reference for the definition is [Wal], section 14.7. Write Π(G) for the set of infinitesimal equivalence classes of irreducible admissible representations of G. We regard Ĝ as a subset of Π(G). It is not difficult to impose on Π(G) a topology making Ĝ a closed subspace, but we will have no need to do so. If for example G = A is a vector group, then  may be identified (topologically) with the real vector space ia∗0 of imaginary-valued linear functionals on the Lie algebra a0 of A. Similarly, Π(A) may be identified with the complex vector space a ∗ of all complex-valued linear functionals on a0. If G = K is compact, then K̂ = Π(K) is a discrete space. The general situation combines the features of these extreme cases. The unitary dual Ĝ is more or less a noncompact real polyhedron (some possible local pathologies are explained after Theorem 2), and Π(G) is more or less a complexification of Ĝ. It is convenient to impose on G the hypotheses of [Green], 0.1.2: essentially that G be a linear group with abelian Cartan subgroups. These hypotheses are satisfied if G is the group of real points of a connected reductive algebraic group defined over R. We fix a maximal compact subgroup K of G, with corresponding Cartan involution θ. By Harish-Chandra’s subquotient theorem, K is a “large compact subgroup of G” in the sense that a fixed irreducible representation τ of K occurs in irreducible unitary representations of G with multiplicity bounded by a constant depending only on τ .